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Foreword 


The  reporting  of  final  measurement  results,  and  the  uncertainties  associated  with 
the  measurement  processes  used  to  obtain  these  results,  has  always  been  and  continues 
to  be  a  source  of  difficulty.  The  three  articles  reprinted  in  this  publication  are  collected 
here  as  a  convenient  reference  source  for  experimenters  who  must  face  the  difficult 
task  of  deciding  how  to  express  measurement  uncertainties.  The  philosophical  basis, 
general  guidelines,  and  specific  recommendations  for  expressing  uncertainties 
contained  within  these  articles  have  evolved  at  NBS  over  a  period  of  many  years. 

The  first  article  originally  appeared  in  Science  in  1968.  This  article  develops  the 
underlying  basis  and  general  guidelines  on  the  forms  of  expression  needed  for 
uncertainty  statements,  and  presents  specific  recommendations  for  four  distinct  cases: 
(i)  when  both  systematic  error  and  imprecision  are  negligible;  (ii)  when  systematic 
error  is  not  negligible,  and  imprecision  is  negligible;  (iii)  when  neither  systematic  error 
nor  imprecision  is  negligible;  and  (iv)  when  systematic  error  is  negligible,  and 
imprecision  is  not  negligible. 

The  second  article,  written  as  a  companion  to  the  first,  originally  appeared  in  a  1968 
issue  of  M&D:  Measurements  and  Data.  It  gives  a  condensed  summary  of  the  recom- 
mendations presented  in  the  first  article,  and  provides  tabular  guides  to  commonly 
used  statements  of  imprecision,  systematic  error,  and  uncertainty. 

The  third  article  is  a  postscript  to  the  two  preceding  articles,  and  was  prepared  in 
1980  for  an  internal  NBS  communications  manual.  It  reinforces  the  major  thrust  and 
content  of  the  earlier  articles,  but  includes  more  recent  thought  particularly  in  regard 
to  overall  uncertainty  statements. 

The  first  two  articles  have  since  been  reprinted  in  several  NBS  publications 
including  Special  Publication  300,  Volume  1,  Precision  Measurement  and  Calibra- 
tion: Statistical  Concepts  and  Procedures  (Harry  H.  Ku,  ed.,  1969).  The  1980  NBS 
communications  manual  incorporated  the  second  and  third  articles,  but  did  not 
reprint  the  first  article.  Furthermore,  this  manual  is  not  accessible  outside  NBS.  This 
special  publication,  therefore,  collects  all  three  articles,  for  the  first  time,  in  one 
convenient  source  which  is  available  to  the  many  scientists  and  engineers  throughout 
the  entire  measurement  community. 


in 


Abstract 


This  publication  reprints  and  collects  in  one  convenient  source  three  articles,  by 
NBS  authors,  that  present  a  philosophical  basis,  general  guidelines,  and  specific 
recommendations  for  expressing  the  uncertainties  of  final  measurement  results. 

Key  words:  accuracy;  errors;  measurement  uncertainty;  precision;  reporting  of 
measurement  data;  systematic  error;  uncertainties. 
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Expression  of  the  Uncertainties 

of  Final  Results 

Clear  statements  of  the  uncertainties  of  reported 
values  are  needed  for  their  critical  evaluation. 

Churchill  Eisenhart 


Measurement  of  some  property  of  a 
thing  in  practice  always  takes  the  form 
of  a  sequence  of  steps  or  operations 
that  yield  as  an  end  result  a  number 
that  serves  to  represent  the  amount  or 
quantity  of  some  particular  property  of 
a  thing — a  number  that  indicates  how 
much  of  this  property  the  thing  has, 
for  someone  to  use  for  a  specific  pur- 
pose. The  end  result  may  be  the  out- 
come of  a  single  reading  of  an  instru- 
ment, with  or  without  corrections  for 
departures  from  prescribed  conditions. 
More  often  it  is  some  kind  of  average, 
for  example,  the  arithmetic  mean  of  a 
number  of  independent  determinations 
of  the  same  magnitude,  or  the  final 
result  of  a  least  squares  "reduction"  of 
measurements  of  a  number  of  different 
magnitudes  that  bear  known  relations 
with  one  another  in  accordance  with  a 
definite  experimental  plan.  In  general, 
the  purpose  for  which  the  answer  is 
needed  determines  the  precision  or  ac- 
curacy required  and  ordinarily  also  the 
method  of  measurement  employed. 

Although  the  accuracy  required  of  a 
reported  value  depends  primarily  on 
the  intended  use,  or  uses,  of  the  value, 
one  should  not  ignore  the  requirements 
of  other  uses  to  which  it  is  likely  to 
be  put.  A  reported  value  whose  accu- 
racy is  entirely  unknown  is  worthless. 

Strictly  speaking,  the  actual  error  of 
a  reported  value,  that  is  the  magnitude 
and  sign  of  its  deviation  from  the  truth 
(1),  is  usually  unknowable.  Limits  to 
this  error,  however,  can  usually  be  in- 
ferred— with  some  risk  of  being  incor- 
rect— from  the  precision  of  the  mea- 
surement process  by  which  the  reported 
value  was  obtained,  and  from  rea- 
sonable limits  to  the  possible  bias  of 
the  measurement  process.  The  bias,  or 
systematic  error,  of  a  measurement  proc- 


ess is  the  magnitude  and  direction  of 
its  tendency  to  measure  something 
other  than  what  was  intended;  its  preci- 
sion refers  to  the  typical  closeness  to- 
gether of  successive  independent  mea- 
surements of  d  single  magnitude  gen- 
erated by  repeated  applications  of  the 
process  under  specified  conditions;  and 
its  accuracy  is  determined  by  the 
closeness  to  the  true  value  characteris- 
tic of  such  measurements. 

Precision  and  accuracy  are  inherent 
characteristics  of  the  measurement  proc- 
ess employed  and  not  of  the  particular 
end  result  obtained.  From  experience 
with  a  particular  measurement  process 
and  knowledge  of  its  sensitivity  to  un- 
controlled factors,  one  can  often  place 
reasonable  bounds  on  its  likely  system- 
atic error  (bias).  It  is  also  necessary  to 
know  how  well  the  particular  value  in 
hand  is  likely  to  agree  with  other 
values  that  the  same  measurement  proc- 
ess might  have  provided  in  this  in- 
stance, or  might  yield  on  remeasure- 
ment  of  the  same  magnitude  on  another 
occasion.  Such  information  is  provided 
by  the  estimated  standard  error 
(2)  of  the  reported  value,  which  mea- 
sures (or  is  an  index  of)  the  charac- 
teristic disagreement  of  repeated  deter- 
minations of  the  same  quantity  by  the 
same  method,  and  thus  serves  to  indi- 
cate the  precision  (strictly,  the  impreci- 
sion) of  the  reported  value  (5). 

Four  Distinct  Forms  of 
Expression  Needed 

The  uncertainty  of  a  reported  value 
is  indicated  by  stating  credible  limits 
to  its  likely  inaccuracy.  No  single 
form  of  expression  for  these  limits  is 
universally  satisfactory.  In  fact,  differ- 


ent forms  of  expression  are  recom- 
mended, which  will  depend  on  the  rela- 
tive magnitudes  of  the  imprecision  and 
likely  bias,  and  their  relative  impor- 
tance in  relation  to  the  intended  use  of 
the  reported  value,  as  well  as  to  other 
possible  uses  to  which  it  may  be  put 
(4). 

Four  distinct  cases  need  to  be  recog- 
nized: (i)  both  systematic  error  and  im- 
precision negligible,  in  relation  to  the 
requirements  of  the  intended  and  likely 
uses  of  the  result;  (ii)  systematic  error 
not  negligible,  imprecision  negligible; 
(iii)  neither  systematic  error  nor  im- 
precision negligible;  and  (iv)  systematic 
error  negligible,  imprecision  not  negli- 
gible. 

Specific  recommendations  with  re- 
spect to  each  of  these  cases  are  made 
below.  General  guidelines  upon  which 
these  specific  recommendations  are 
based  are  discussed  in  the  following 
paragraphs. 

Perils  of  Shorthand  Expressions 

Final  results  and  their  respective  un- 
certainties should  be  reported  in  sen- 
tence form  whenever  possible.  The 
shorthand  form  "a  ±  b"  should  be 
avoided  in  abstracts  and  summaries;  and 
never  used  without  explicit  explana- 
tion of  its  connotation.  If  no  explana- 
tion is  given,  many  persons  will  take 
±b  to  signify  bounds  to  the  inaccuracy 
of  a.  Others  may  assume  that  b  is  the 
"standard  error,"  or  the  "probable  er- 
ror." of  a,  and  hence  the  uncertainty 
of  a  is  at  least  ±3b,  or  ±4b,  respectively. 
Still  others  may  take  b  to  be  an  indica- 
tion merely  of  the  imprecision  of  the  in- 
dividual measurements,  that  is,  to  be  the 
"standard  deviation,"  or  the  "average 
deviation,"  or  the  "probable  error"  of 
a  single  observation.  Each  of  these  in- 
terpretations reflects  a  practice  of  which 
instances  can  be  found  in  current 
scientific  literature.  As  a  step  in  the 
direction  of  reducing  this  current  con- 
fusion, it  is  recommended  that  the  use 
of  "a  ±  b"  in  presenting  results  be 
limited  to  that  sanctioned  for  the  case 
of  tabular  results  in  the  fourth  recom- 
mendation of  the  section  below  headed 
"Systematic  error  not  negligible,  im- 
precision negligible." 


The  author  is  a  senior  research  fellow  and 
former  chief  of  the  Statistical  Engineering  Labora- 
tory at  the  National  Bureau  of  Standards,  Wash- 
ington. D.C.  20234.  The  recommendations  pre- 
sented in  this  paper  hare  evolved  at  the  Bureau 
over  a  period  of  many  years  and  are  made 
public  here  for  general  information,  and  to  educe 
comments  and  suggestions. 
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Imprecision  and  Systematic  Error 
Require  Separate  Treatment 

Since  imprecision  and  systematic 
error  are  distinctly  different  components 
of  inaccuracy,  and  are  subject  to  dif- 
ferent treatments  and  interpretations  in 
usage,  two  numerics  respectively  ex- 
pressing the  imprecision  and  bounds 
to  the  systematic  error  of  the  reported 
result  should  be  used  whenever  both 
of  these  errors  are  factors  requiring 
consideration.  Such  instances  are  dis- 
cussed in  the  section  below  for  the  case 
of  "Neither  systematic  error  nor  im- 
precision negligible." 

In  quoting  a  reported  value  and  its 
associated  uncertainty  from  the  litera- 
ture, the  interpretation  of  the  uncer- 
tainty quoted  should  be  stated  if  given 
by  the  author.  If  the  interpretation  is 
not  known,  a  remark  to  this  effect 
is  in  order.  This  practice  may  induce 
authors  to  use  more  explicit  formula- 
tions of  their  statements  of  uncertainty. 

Standard  Deviation  and  Standard  Error 

The  terms  standard  deviation  and 
standard  error  should  be  reserved  to 
denote  the  canonical  values  for  the 
measurement  process,  based  on  consid- 
erable recent  experience  with  the  mea- 
surement process  or  processes  involved. 
When  there  is  insufficient  recent  ex- 
perience, an  estimate  of  the  standard 
error  (standard  deviation)  must  of  ne- 
cessity be  computed  by  recognized  sta- 
tistical procedures  from  the  same  mea- 
surements as  the  reported  value  itself. 
To  avoid  possible  misunderstanding,  in 
such  cases,  the  term  "computed  (or 
estimated)  standard  error"  ("computed 
standard  deviation")  should  be  used.  A 
formula  for  calculating  this  computed 
standard  error  is  given  in  the  section 
below  for  the  case  of  "Neither  system- 
atic error  nor  imprecision  negligible." 

Uncertainties  of  Accepted  Values  of 
Fundamental  Constants  or 
Primary  Standards 

If  the  uncertainty  in  the  accepted 
value  of  a  national  primary  standard  or 
of  some  fundamental  constant  of  na- 
ture (for  example,  in  the  volt  as  main- 
tained at  the  National  Bureau  of  Stan- 
dards, or  in  the  acceleration  of  gravity 
g  on  the  Potsdam  basis)  is  an  important 
source  of  systematic  error  affecting  the 
measurement  process,  no  allowance  for 


possible  systematic  error  from  this 
source  should  be  included  ordinarily 
in  evaluating  overall  bounds  to  the  sys- 
tematic error  of  the  measurement  proc- 
ess. Since  the  error  concerned,  what- 
ever it  is,  affects  all  results  obtained 
by  the  method  of  measurement  in- 
volved, to  include  an  allowance  for  this 
error  would  be  to  make  everybody's 
results  appear  unduly  inaccurate  rela- 
tive to  each  other.  In  such  instances 
one  should  state:  (i)  that  measurements 
obtained  by  the  process  concerned  are 
expressed  in  terms  of  the  volt  (or  the 
kilogram,  or  other  unit)  "as  maintained 
at  the  National  Bureau  of  Standards," 
or  (ii)  that  the  indicated  bounds  to  the 
systematic  error  of  the  process  are  ex- 
clusive of  the  uncertainty  of  the  stated 
value  adopted  for  some  particular  con- 
stant or  quantity.  An  example  of  the 
latter  form  of  statement  is: 

.  .  .  neglecting  the  uncertainty  of  the  value 
6.6256  x  10 ~M  joule  seconds  adopted  for 
Planck's  constant. 

Systematic  Error  and  Imprecision 
Both  Negligible 

In  this  case  the  reported  result 
should  be  given,  after  rounding,  to  the 
number  of  significant  figures  consist- 
ent with  the  accuracy  requirements  of 
the  situation,  together  with  an  explicit 
statement  of  its  accuracy.  An  example 
is: 

.  .  .  the  wavelengths  of  the  principal  visible 
lines  of  mercury-198  have  been  measured 
relative  to  the  6057.802106  A  (angstrom 
units)  line  of  krypton-98,  and  their  values 
in  vacuum  are 

5792.2685  A 
5771.1984  A 
5462.2706  A 
4359.5625  A 
4047.7146  A 
correct  to  eight  significant  figures. 

It  needs  to  be  emphasized  that  if  no 
statement  of  accuracy  or  precision  ac- 
companies a  reported  number,  then,  in 
accordance  with  the  usual  conventions 
governing  rounding,  this  number  will 
ordinarily  be  interpreted  as  being  ac- 
curate within  ±Vi  unit  in  the  last  signif- 
icant figure  given;  that  is,  it  will  be 
understood  that  its  inaccuracy  before 
rounding  was  less  than  ±  5  units  in  the 
next  place.  The  statement  "correct  to 
eight  significant  figures"  is  included  ex- 
plicitly in  the  foregoing  example,  rather 
than  left  to  be  understood  in  order  to 
forestall  any  concern  that  an  explicit 
statement  of  lesser  accuracy  was  in- 
advertently omitted. 


Systematic  Error  Not  Negligible, 
Imprecision  Negligible 

When  the  imprecision  of  a  result  is 
negligible,  but  the  inherent  systematic 
error  of  the  measurement  process  con- 
cerned is  not  negligible,  then  the  fol- 
lowing rules  are  recommended: 

1)  Qualification  of  a  reported  result 
should  be  limited  to  a  single  quasi- 
absolute  type  of  statement  that  places 
bounds  on  its  inaccuracy. 

2)  These  bounds  should  be  stated  to 
no  more  than  two  significant  figures. 

3 )  The  reported  result  itself  should 
be  given  (that  is,  rounded)  to  the  last 
place  affected  by  the  stated  bounds 
(unless  it  is  desired  to  indicate  and 
preserve  such  relative  accuracy  or  pre- 
cision of  a  higher  order  that  it  may 
possess  for  certain  particular  uses). 

4)  Accuracy  statements  should  be 
given  in  sentence  form  in  all  cases, 
except  when  a  number  of  results  of 
different  accuracies  are  presented,  for 
example,  in  tabular  arrangement.  If  it 
is  necessary  or  desirable  to  indicate 
the  respective  accuracies  of  a  number 
of  results,  the  results  should  be  given 
in  the  form  a  ±  b  (or  a  t?c,  if  neces- 
sary) with  an  appropriate  explanatory 
remark  (as  a  footnote  to  the  table, 
or  incorporated  in  the  accompanying 
text)  to  the  effect  that  the  ±b,  or  ±bc, 
signify  bounds  to  the  systematic  errors 
to  which  the  a's  may  be  subject. 

5)  The  fact  that  the  imprecision  is 
negligible  should  be  stated  explicitly. 

The  particular  form  of  the  quasi- 
absolute  type  of  statement  employed 
in  a  given  instance  will  depend  ordi- 
narily on  personal  taste,  experience, 
current  and  past  practice  in  the  field 
of  activity  concerned,  and  so  forth. 
Some  examples  of  good  practice  are: 

.  .  .  is  (are)  not  in  error  by  more  than  1 
part  in  (x). 

.  .  .  is  (are)  accurate  within  ±  (x  units) 
[or  ±  (x)  percent], 

.  .  .  is  (are)  believed  accurate  within 
(  ). 

Positive  wording,  as  in  the  first  two 
of  these  quasi-absolute  statements,  is 
appropriate  only  when  the  stated 
bounds  to  the  possible  inaccuracy  of 
the  reported  value  are  themselves  relia- 
bly established.  However,  when  the  in- 
dicated bounds  are  somewhat  conjec- 
tural, it  is  desirable  to  signify  this 
fact  (and  put  the  reader  on  guard)  by 
inclusion  of  some  modifying  expres- 
sion such  as  "believed,"  "considered," 
"estimated  to  be,"  "thought  to  be,"  and 
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so  forth,  as  exemplified  by  the  third  of 
the  foregoing  examples. 

The  term  uncertainty  may  sometimes 
be  used  effectively  to  achieve  a  concise- 
ness of  expression  otherwise  difficult  or 
impossible  to  attain.  Thus,  one  might 
make  a  statement  such  as: 

The  uncertainties  in  the  above  values 
are  not  more  than  ±  0.5 °C  in  the  range 
0°C  to  1100°C,  and  then  increase  to  ± 
2°C  at  1450°C, 

or 

The  uncertainty  in  this  value  does  not 
exceed  .  .  .  excluding  (or,  including)  the 
uncertainty  of  ...  in  the  value  .  .  .  adopted 
for  the  (reference  standard  involved). 

A  statement  giving  numerical  limits 
of  uncertainty  as  in  the  above  should 
be  followed  by  a  brief  discussion  tell- 
ing how  the  limits  were  derived. 

Finally,  the  following  forms  of  quasi- 
absolute  statements  are  considered  poor 
practice,  and  are  to  be  avoided: 


The  accuracy  of 
The  accuracy  of 


.  is  5  percent. 
.  is  ±  2  percent. 


These  are  presumably  intended  to 
mean  that  the  result  concerned  is  not 
inaccurate,  that  is,  not  in  error,  by 
more  than  5  percent  or  2  percent,  re- 
spectively, but  they  explicitly  state  the 
opposite. 


Neither  Systematic  Error  Nor 
Imprecision  Negligible 

When  neither  the  imprecision  nor  the 
systematic  error  of  a  result  are  negligi- 
ble, then  the  following  rules  are  rec- 
ommended: 

1)  A  reported  result  should  be  quali- 
fied by  a  quasi-absolute  type  of  state- 
ment that  places  bounds  on  its  sys- 
tematic error,  and  a  separate  statement 
of  its  standard  error  or  its  probable 
error,  or  of  an  upper  bound  thereto, 
whenever  a  reliable  determination  of 
such  value  or  bound  is  available.  Other- 
wise a  computed  value  of  the  standard 
error,  or,  probable  error,  so  designated, 
should  be  given  together  with  a  state- 
ment of  the  number  of  degrees  of 
freedom  on  which  it  is  based. 

2)  The  bounds  to  its  systematic  error 
and  the  measure  of  its  imprecision 
should  be  stated  to  no  more  than  two 
significant  figures. 

3)  The  reported  result  itself  should 
be  stated  at  most  to  the  last  place  af- 
fected by  the  finer  of  the  two  qualify- 
ing statements  (unless  it  is  desired  to 
indicate  and  preserve  such  relative  ac- 
curacy or  precision  of  a  higher  order 


that  it  may  possess  for  certain  particu- 
lar uses). 

4)  The  qualification  of  a  reported 
result  with  respect  to  its  imprecision 
and  systematic  error  should  be  given 
in  sentence  form,  except  when  results 
of  different  precision  or  with  different 
bounds  to  their  systematic  errors  are 
presented  in  tabular  arrangement.  If 
it  is  necessary  or  desirable  to  indicate 
their  respective  imprecisions  or  bounds 
to  their  respective  systematic  errors, 
such  information  may  'be  given  in  a 
parallel  column  or  columns,  with  ap- 
propriate identification. 

Here,  and  in  the  next  section,  the 
term  standard  error  is  to  be  under- 
stood as  signifying  the  standard  devia- 
tion of  the  reported  value  itself,  not  as 
signifying  the  standard  deviation  of  the 
single  determination  (unless,  of  course, 
the  reported  value  is  simply  the  result 
of  a  single  determination). 

The  above  recommendations  should 
not  be  construed  to  exclude  the  pres- 
entation of  a  quasi-abolute  type  of  state- 
ment placing  bounds  on  the  inaccuracy, 
that  is,  on  the  overall  uncertainty,  of  a 
reported  value,  provided  that  separate 
statements  of  its  imprecision  and  its 
possible  systematic  error  are  included 
also.  To  be  in  good  taste,  the  bounds 
indicating  the  overall  uncertainty 
should  not  be  numerically  less  than  the 
corresponding  bounds  placed  on  the 
systematic  error  outwardly  increased  by 
at  least  three  times  the  standard  error. 
The  fourth  of  the  following  examples 
of  good  practice  is  an  instance  at 
point: 

The  standard  errors  of  these  values  do 
not  exceed  0.000004  inch,  and  their  sys- 
tematic errors  are  not  in  excess  of  0.00002 
inch. 

The  standard  errors  of  these  values  are 
less  than  (x  units),  and  their  systematic  er- 
rors are  thought  to  be  less  than  ±  (y 
units).  No  additional  uncertainty  is  as- 
signed for  the  conversion  to  the  chemical 
scale  since  the  adopted  conversion  factor 
is  taken  as  1.000275  exactly. 

.  .  .  with  a  standard  error  of  (x  units), 
and  a  systematic  error  of  not  more  than 
±  (y  units). 

.  .  .  with  an  overall  uncertainty  of  ±  3 
percent  based  on  a  standard  error  of  0.5 
percent  and  an  allowance  of  ±  1.5  percent 
for  systematic  error. 

When  a  reliably  established  value  for 
the  relevant  standard  error  is  available, 
and  the  dispersion  of  the  present  mea- 
surements is  in  keeping  with  this  ex- 
perience, then  this  canonical  value  of 
the  standard  error  should  be  used  (5). 
If  such  experience  indicates  that  the 
standard  error  is  subject  to  fluctuations 


greater  than  the  intrinsic  variation  of 
such  a  measure,  then  an  appropriate 
upper  bound  should  be  given,  for  ex- 
ample, as  in  the  first  tv/o  of  the  above 
examples,  or  by  changing  "a  standard 
error  .  .  ."  in  the  third  and  fourth 
examples  to  "an  upper  bound  to  the 
standard  error  .  .  ." 

When  there  is  insufficient  recent 
experience  with  the  measurement  proc- 
esses involved,  an  estimate  of  the 
standard  error  must  of  necessity  be 
computed  by  recognized  statistical  pro- 
cedures from  the  same  measurements 
as  the  reported  value  itself.  It  is 
essential  that  such  computations  be 
carried  out  according  to  an  agreed- 
upon  standard  procedure,  and  the  results 
thereof  presented  in  sufficient  detail  to 
enable  the  reader  to  form  his  own  judg- 
ment, and  make  his  own  allowances 
for  their  inherent  uncertainties.  Tc 
avoid  possible  misunderstanding,  in  such 
cases,  first,  the  term  computed  standard 
error  should  be  used;  second,  the  esti- 
mate of  the  standard  error  employed 
should  be  that  obtained  from 

estimate  of  standard  error  = 

^  sum  of  squared  residuals 

where  n  is  the  (effective)  number  of 
completely  independent  determinations 
of  which  a  is  the  arithmetic  mean  (or 
other  appropriate  least-squares  adjusted 
value)  and  v  is  the  number  of  degrees 
of  freedom  involved  in  the  sum  of 
squared  residuals  (that  is,  the  number 
of  residuals  minus  the  number  of  fitted 
constants  or  other  independent  con- 
straints on  the  residuals);  and  third,  the 
number  of  degrees  of  freedom  should 
be  explicitly  stated.  If  the  reported 
value  a  is  the  arithmetic  mean,  then: 

estimate  of  standard  error  =  CsVn)% 
where 


r  =  2  (x,  -  af/(n  -  1) 
i=l 


and  n  is  the  number  of  completely  in- 
dependent determinations  of  which  a  is 
the  arithmetic  mean.  For  example: 

.  .  .  which  is  the  arithmetic  mean  of  (n) 
independent  determinations  and  has  a  stan- 
dard error  of  .  .  . 

.  .  .  with  an  overall  uncertainty  of 
±  5.2  km/sec  based  on  a  standard  error 
of  1.5  km/sec  and  estimated  bounds  of 
±  0.7  km/sec  on  the  systematic  error. 
(The  figure  5.2  is  equal  to  0.7  plus  3 
times  1.5.) 

or,  if  based  on  a  computed  standard 
error, 

The  computed  probable  error  (or,  stan- 
dard error)  of  these  values  is  (x  units), 
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based  on  (c)  degrees  of  freedom,  and  the 
systematic  error  is  estimated  to  be  less  than 
±  (y  units). 

.  .  .  with  an  overall  uncertainty  of  ±  7 
km/sec  derived  from  bounds  of  ±  0.7 
km/sec  on  the  systematic  error  and  a  com- 
puted standard  error  of  1.5  km/sec  based 
on  9  degrees  of  freedom.  [The  number 
7  is  approximately  equal  to  0.7  +  (4.3  X 
1.5),  where  4.3  is  the  value  of  Student's  t 
for  9  degrees  of  freedom  exceeded  in  ab- 
solute value  with  0.002  probability.  As 
f-*  oo,  f.oos  («-)  —  3.090.] 

When  the  reported  value  is  the  result 
of  a  complex  measurement  process 
and  is  obtained  as  a  function  of  sev- 
eral quantities  whose  standard  errors 
have  been  computed,  these  several 
quantities  and  their  standard  errors 
should  usually  be  reported,  together 
with  a  description  of  the  method  of 
computation  by  which  the  standard 
errors  were  combined  to  provide  an 
overall  estimate  of  imprecision  for  the 
reported  value. 

Systematic  Error  Negligible, 
Imprecision  Not  Negligible 

When  the  systematic  error  of  a  result 
is  negligible  but  its  imprecision  is  not, 
the  following  rules  are  recommended: 

1)  Qualification  of  a  reported  value 
should  be  limited  to  a  statement  of  its 
standard  error  or  of  an  upper  bound 
thereto,  whenever  a  reliable  determina- 
tion of  such  value  or  bound  is  avail- 
able. Otherwise  a  computed  value  of 
the  standard  error,  so  designated, 
should  be  given  together  with  a  state- 
ment of  the  number  of  degrees  of 
freedom  on  which  it  is  based. 

2)  The  standard  error  or  upper 
bound  thereto,  should  be  stated  to  not 
more  than  two  significant  figures. 

3)  The  reported  result  itself  should 
be  stated  at  most  to  the  last  place  af- 
fected by  the  stated  value  or  bound 
to  its  imprecision  (unless  it  is  desired 
to  indicate  and  preserve  such  relative 
precision  of  a  higher  order  that  it  may 
possess  for  certain  particular  uses). 

4)  The  qualification  of  a  reported 
result  with  respect  to  its  imprecision 
should  be  given  in  sentence  form,  ex- 
cept when  results  of  different  precision 
are  presented  in  tabular  arrangement 
and  it  is  necessary  or  desirable  to  indi- 
cate their  respective  imprecisions  in 
which  event  such  information  may  be 
given  in  a  parallel  column  or  columns, 
with  appropriate  identification. 

5)  The  fact  that  the  systematic  er- 
ror is  negligible  should  be  stated  ex- 
plicitly. 

The  above  recommendations  should 
not  be  construed  to  exclude  the  pres- 


entation of  a  quasi-absolute  type  of 
statement  placing  bounds  on  its  possible 
inaccuracy,  provided  that  a  separate 
statement  of  its  imprecision  is  included 
also.  To  be  in  good  taste,  such  bounds 
to  its  inaccuracy  should  be  numeri- 
cally equal  to  at  least  three  times  the 
stated  standard  error.  The  fourth  of 
the  following  examples  of  good  practice 
is  an  instance  at  point. 

The  standard  errors  of  these  values  are 
less  than  (x  units). 
.  .  .  with  a  standard  error  of  (x  units). 

.  .  .  with  a  computed  standard  error  of 
(x  units)  based  on  (v)  degrees  of  freedom. 

.  .  .  with  an  overall  uncertainty  of  ±  4.5 
km/sec  derived  from  a  standard  error  of 
1.5  km/sec.  (The  figure  4.5  is  equal  to 
3  X  1.5.) 

or,  if  based  on  a  computed  standard 
error, 

.  .  .  with  an  overall  uncertainty  of  ±  6.5 
km/sec  derived  from  a  computed  standard 
error  of  1.5  km/sec  (based  on  9  degrees 
of  freedom).  (The  number  6.5  is  equal  to 
4.3  x  1.5,  where  4.3  is  the  value  of  Stu- 
dent's t  for  9  degrees  of  freedom  ex- 
ceeded in  absolute  value  with  0.002  prob- 
ability. As        co,  tJm  (v)  3.090.) 

The  remarks  with  regard  to  a  com- 
puted standard  error  in  the  preceding 
section  apply  with  equal  force  to  the 
last  two  examples  above. 


Conclusion 

The  foregoing  recommendations  call 
for  fuller  and  sharper  detail  than  is 
general  in  common  pactice.  They 
should  be  regarded  as  minimum  stan- 
dards of  good  practice.  Of  course,  many 
instances  require  fuller  treatment  than 
that  recommended  here. 

Thus,  in  the  case  of  determinations 
of  the  "fundamental  physical  con- 
stants" and  other  basic  properties  of 
nature,  the  author  or  authors  should 
give  a  detailed  account  of  the  various 
components  of  imprecision  and  sys- 
tematic error,  and  list  their  respective 
individual  magnitudes  in  tabular  form, 
so  that  (i)  the  state  of  the  art  will  be 
more  clearly  revealed,  (ii)  each  individ- 
ual user  of  the  final  result  may  decide 
for  himself  which  of  the  indicated  com- 
ponents of  imprecision  or  systematic 
error  are,  or  are  not,  relevant  to  his 
use  of  the  final  result,  and  (iii) — most 
important — the  final  result  itself  or  its 
uncertainty  can  be  modified  appropri- 
ately in  the  light  of  later  advances.  This 
is,  and  has  long  been,  the  practice  fol- 
lowed in  the  best  reports  of  funda- 
mental studies,  but  current  efforts  to 


prepare  critically  evaluated  standard 
reference  data  have  revealed  that  far 
too  great  a  fraction  of  the  data  in  the 
scientific,  literature  "cannot  be  criti- 
cally evaluated  because  the  minimum 
of  essential  information  is  not  present" 
(6). 
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EXPRESSIONS  OF  IMPRECISION, 

SYSTEMATIC  ERROR, 
AND 

UNCERTAINTY  ASSOCIATED 
WITH  A  REPORTED  VALUE 

HARRY  H.  KU,  National  Bureau  of  Standards 


The  work  of  a  calibration  laboratory 
may  be  thought  of  as  a  sequence  of 
operations  that  result  in  the  collection,  stor- 
age, and  transmittal  of  information.  In  mak- 
ing a  statement  of  uncertainty  of  the  result 
of  calibration,  the  calibration  laboratory 
transmits  information  to  its  clients  on  the 
particular  item  calibrated. 

It  is  logical,  then,  to  require  the  trans- 
mitted information  to  be  meaningful  and 
unambiguous,  and  to  contain  all  the  rele- 
vant information  in  the  possession  of  the 
laboratory.  The  information  content  of  the 
statement  of  uncertainty  determines,  to  a 
large  extent,  the  worth  of  the  calibrated 
value. 

A  common  deficiency  in  many  statements 
of  uncertainty  is  that  they  do  not  convey 
all  the  information  a  calibration  laboratory 
has  to  offer,  information  acquired  through 
much  ingenuity  and  hard  work.  This  defi- 
ciency usually  originates  in  two  ways: 

1.  Loss  of  information  through  oversim- 
plification, and 

2.  loss  of  information  through  the  inability 
of  the  laboratory  to  take  into  account  in- 
formation accumulated  from  its  past  ex- 
perience. 

With  the  increasingly  stringent  demands 
for  improved  precision  and  accuracy  of  cali- 
bration work,  calibration  laboratories  as  a 
whole  just  cannot  afford  such  luxury. 

Traceability  to  the  national  standards, 
accuracy  ratios,  and  class  tolerance  require- 
ments are  simplified  concepts  that  aim  to 
achieve  different  degrees  of  accuracy  re- 
quirements. These  concepts  and  the  result- 


ing statements  are  useful  on  certain  occa- 
sions, but  fail  whenever  the  demand  is 
exacting.  The  general  practice  of  obliterat- 
ing all  the  identifiable  components  of  un- 
certainty, by  combining  them  into  an  over- 
all uncertainty,  just  for  the  sake  of  simplicity, 
is  another  case  in  point.  After  all,  if  the 
calibration  laboratory  reports  all  the  per- 
tinent information  in  separate  components, 
the  user  can  always  combine  them  or  use 
them  individually,  as  he  sees  fit.  On  the 
other  hand,  if  the  user  is  given  only  one 
number,  he  can  never  disentangle  this  num- 
ber into  its  various  components.  Since  the 
information  buried  under  these  oversimpli- 
fied statements  is  available,  and  may  well 
be  useful  to  sophisticated  customers,  such 
practices  result  in  substantial  waste  of  ef- 
fort and  resources. 

In  calibrating  an  item  by  repeating  the 
same  calibration  procedure,  the  calibration 
laboratory  gains  increments  of  information 
about  its  calibration  system.  These  incre- 
ments of  information  are  quantified  and  ac- 
cumulated for  the  benefit  of  the  calibration 
laboratory.  If  the  precision  of  the  calibra- 
tion process  remains  unchanged,  the  sta- 
tistical measure  of  dispersion  (s)  -  i.e., 
the  standard  deviations  computed  from  these 
sets  of  data  -  can  be  pooled  together, 
weighted  by  their  respective  degrees  of 
freedom.  When  many  such  increments  of 
information  are  combined,  an  accepted  or 
canonical  value  of  standard  deviation  (  <r ) 
is  established.  This  established  (canonical) 
value  of  standard  deviation  characterizes 
the  precision  of  the  calibration  process,  and 
is  treasured  information  in  any  calibration 
laboratory. 
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Hence,  the  canonical  value  of  standard 
deviation  is  the  quantification  of  informa- 
tion accumulated  from  past  experiences  of 
the  calibration  laboratory,  and  is  an  essen- 
tial element  of  the  statement  of  uncertainty. 
The  standard  deviation  (s)  computed  from 
the  current  calibration  is  used  to  check  the 
precision  of  current  work,  and  to  add  to  the 
pool  of  information  on  the  process,  but  cer- 
tainly does  not  represent  all  the  informa- 
tion available  in  the  possession  of  an  es- 
tablished calibration  laboratory.  Only  by 
passing  its  accumulated  information  to  the 
users  is  the  calibration  laboratory  perform- 
ing a  complete  service. 

STATEMENT  OF  UNCERTAINTY 

In  the  preparation  of  a  statement 
of  uncertainty,  it  is  helpful  to  bear  in  mind 
that: 

1.  The  derivation  of  a  statement  of  un- 
certainty has  as  its  foundation  the  work 
done  in  the  laboratory,  and  is  based  on 
information  accumulated  from  past  experi- 
ence, and 

2.  In  general,  information  is  lost  through 
oversimplification,    and   demands   for  im- 


proved precision  and  accuracy  cannot  be 
met  with  simplified  statements  of  uncer- 
tainty. 

Unless  a  statement  of  uncertainty  is  well 
formulated  and  supported,  it  is  difficult  to 
say  what  is  meant  by  the  statement,  a  dif- 
ficulty frequently  encountered.  Since  the 
evaluation  of  uncertainty  is  part  and  par- 
cel of  the  high  standard  of  work  of  a  cali- 
bration laboratory,  the  statement  of  uncer- 
tainty deserves  all  the  attention  required 
to  make  the  statement  both  realistic  and 
useful.  To  this  end,  Tables  1,  2  and  3  give 
terms  and  expressions  compiled  as  a  ready 
reference  for  those  who  are  searching  for 
some  appropriate  format  or  wording,  to  car- 
ry out  the  thoughts  expressed.  They  sum- 
marize the  recommended  practices  on 
expression  of  uncertainties  as  given  in 
Chapter  23  of  NBS  Handbook  91.  A  re- 
vised version  of  this  chapter  with  the  title 
"Expression  of  Uncertainties  of  Final  Re- 
sults" by  Churchill  Eisenhart  may  be  found 
in  NBS  Special  Publication  300-1.  Figure  1 
gives  a  condensed  summary  of  this 
material.  Tables  1,  2,  and  3  give  details  of 
forms  of  imprecision,  systematic  error, 
and  uncertainty  statements. 
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TABLE  1  -  IMPRECISION  STATEMENTS 


Value 
reported 

Index  or  Measure  of  Error 

Remarks 

Precision  of  a  mea- 
surement (calibra- 
tion)  process 

(a).  Standard    deviation   [u]  of 
a  single     determination  (ob- 
servation} 

f7  (or  s  with  the  associated  degrees  of  freedom1)  is  of 
main   interest  as   an    index   of  precision  of  the  mea- 
surement process.  If  the  average  of  n  such  measure- 
ments is  also  reported,  see  (b)  below. 

A  r  1 1  h  m  e  1 1  c     m  eon 
(in)   °'    n  numbers 

(b)    Standard  error  (rx  / ~^^/  n )  of  the 
reported  value 

n  is  of  m oin  interest'   the  nu m ber  n  is  o  1  s o  essentiol 
information;  cassumed  known.  ' 

(c)  .  2  sigma  limits 

(d)  .  3  sigma  limits 

Commonly  used  bounds  of  imprecisions;  usually  used 
when  a  known,  or  when  n  large. 

(e).  Confidence  interval  (indicate 
one-  or  two-sided) 

Data  points  assumed  to  be  normally  distributed;  report 
confidence  coefficient  (level)  100  (1  —  c^)%  2 

(f).  Half-width  of  confidence  inter- 
val (or  confidence  limits) 

Same  as  (e)  above;  for  symmetrical  two-sided  intervals; 
an  index  to  bounds  of  imprecision  2 

(g).  Probable  error  of  the  reported 
value 

Probable  error  —  .6745  \/n  r^*r  normally  distributed 
data  points  when  <j  known.  Use  of  o/\/n  preferred. 
Incorrect  if  a  not  known. 

(h).  Mean  deviation,  or  average 
deviation,  of  a  measurement  from 
the   mean  calculated  from  the 
sample 

Limiting  mean  of  mean  deviation  =  \  f^jc  n/^Ft'  ^  'or 
normally  distributed  data  points  when  <7  known 
Use  of  a  usually  preferred. 

(i).  Any  of  the  above  expressed 
in  percent,  or  ppm  of  xn- 

State  what  is  being  expressed  in  percent,  eg.,  (o'/y/'n) 
(100/in  ),in  being  a  fairly  constant  value. 

m  means  each  com- 
puted from   n  mea- 
surements 

(i).  (b),  (c),  (d)  and  (f)  above 

If  the  measurements  are  of  equal  precision  and  a  un- 
known, use 

Sr,2  =—    2     s'|3  as  estimate  of  a7.  The  no.   of  de- 
P       m  , 
i  =  l 

grees  of  freedom  associated  with  sp  is  m(n-l). 

(k)   Sample  coefficient  of  variation 

(v  =  4-) 

or  relative  percent 

(v  X  100) 

Appropriate  when  the  m  means  cover  a  wide  range 
and  where  the  v's  computed  for  the  m  sets  are  about 
the   same   magnitude.   Give   range   of   v's  for  the  m 
sets.  The  means  must  be  positive  and  bounded  away 
from  zero. 

Weighted  mean 

w,i,  +w,i3 

%      w,  +  w, 

(1).  Standard  error  (r/  =  2)  of  the 
weighted  mean 

If  w  =   1/rji  2  and  w,    =   1  /cr  =.  J,  then  c5J=  — r~ 

1             '      X  j                       2                1      X  2                                    W,  -f-Wj 

Not  recommended  when  the  cr's  are  not  known  and 
are  estimated  by  s  computed  from  small  number  of 
measurements. 

An  equation  (theo- 
retical or  empirical) 
fitted  to  data  points 
by    the   method  of 
least  squares 

( m )   Standard  deviationco m pu ted 
from  the  deviations  (residuals)  of 
data  points  from  the  fitted  curve 

Report  n    the  number  of  data  points  and  k   the  num- 
ber of  constants  fitted, 

s2=Z    (y,  -y,)7(n-k), 

i=l 

where  y-           is  the  value  on  the  fitted  curve  for  the 
particular  X,.3  Value  of  s  usually  given  in  computer 
print-out. 

Constants  (coeffi- 
cients) in  the  equa- 
tion  fitted   to  the 
data  points  by  the 
method    of  least 
squares 

(n).  Standard  errors  of  the  coeffi- 
cients based  on  the  standard  de- 
viation computed  under  (m) 

Standard  errors   usually  given  in  computer  print-out. 
Report  n  and  k  as  above.3- 
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TABLE  1  -  IMPRECISION  STATEMENTS  -  (Continued) 


Value 
reported 

Index  or  Measure  of  Error 

Remarks 

A  predicted  point  on 
the  curve   y  for  a 
particular  xc 

(o).  Standard  error  (s^)  of  the  pre- 
dicted point 

For  the  straight  line  case,  the  computer  print-out  gives 
the  variance-covariance  matrix  (       s  ). 
sy'  —   S, ,  -f-  2s,2I0  +    s22Xo!  ■  3 
Report  n  and  k. 

A  predicted  observ 
ed  value  for  a  par- 
ticular X0 

(p).  Standard  error  of  the  predict- 
ed value  of  y 

For   the  straight  line  case,  Sy  =  Sy    +s'    where  SyJ 
and    sJ    are   that  given   in  (o)  and  (m)  respectively.3 
Report  n  and  k 

Value  of  function  of 
the  arithmetic 
means  of  several 
measured  variables 

(q).  Standard  error  calculated  by 
the  use  of  propagation  of  error 
formulas 

Appropriate  when  errors  of  measurements  are  small 
compared   to  the  values  of  variables  measured.  Use 
standard  error  ot  the  means  ot  the  variables  in  the 
formulas.  4     Report    number   of    measurements  from 
which  these  standard  errors  are  computed 

Percentage  or  pro- 
portion ( r/  n),  r  and 
n  being  counts 

(r).  Confidence  limits  of  the  true 
proportion  P 

Procedures  for  obtaining  exact  and  approximate  confi- 
dence limits  are  discussed  in  Chapter  7,  NBS  Hand- 
book 91  .  State  one-sided  or  two-sided. 

TABLE  2 -SYSTEMATIC  ERROR  5  (BIAS)  STATEMENTS 

Value 
reported 

Index  or  Measure  of  Error 

Remarks 

Numerical  value  re- 
sulting from  a  mea- 
surement process 

Reasonable  bounds   ascribed  to 
the  value  originating  from: 
(i).  systematic  error   reliably  es- 
tablished 

Detailed  discussions   of  systematic  errors  are  always 
helpful. 

Positive  wording  is  appropriate: 

".  .  .  is  not  in  error  by  more  than  .  .  ." 

".  .  .  is  accurate  within  ±  .  .  ." 

(ii).  systematic  error  estimated 
from  experience  or  by  judgment 

Use  modifier  such  as  "believed",  "estimated",  "consider- 
ed", to  signify  the  conjectural  nature  of  the  statement. 

(iii).  combination  of  a  number  of 
elemental  systematic  errors 

State  explicitly   the   method   of  combination   such  as 
"the  simple  sum  of  the  bounds"  or  "the  square  root  of 
the  sum  of  squares". 

(iv).  uncertainty  in  some  funda- 
mental constant 

Give  reference  to  the  value  of  constant  used. 

(v).  uncertainty  in  calibrated 
values 

Ascertain  the  meaning  of  the  systematic  and  random 
components  of   the   uncertainty   from   the  calibration 
laboratory  so  that  decisions  on  the  uses  of  these  com- 
ponents can  be  made  from  the  correct  interpretations. 

(vi).  bias  in  the  method  of  com- 
putation 

Correct  if  feasible,  or  give  the  magnitudes 
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TABLE  3  -  UNCERTAINTY  STATEMENTS 


Value 
reported 

Index  or  Measure  of  error 

Remarks 

Numerical  value  re- 
sulting from  a  mea- 
surement process 

Bounds  to  inaccuracy: 
(1).  Systematic  error  and  impre- 
cision both  negligible 

Explicit  expression  of  correctness  to  the  last  significant 
figure,  interpreted  as  being  accurate  within  ±l/2  units 
in  the  last  significant  figure  given. 

(2).  Imprecision  negligible.  Bounds 
on  inaccuracy  given  to  no  more 
than  two  significant  figures. 

Sentence  form  preferred  such  as  given  under  remark 
for  (i)  and  (ii).  Footnote  needed  if  bounds  are  given 
in  tabular  form. 

(3).  Systematic    error  negligible. 
Index  of  precision  (b),  (g),  (h),  (i), 
(k),  or  (n)  stated  to  no  more  than 
two  significant  figures 

State  explicitly  the  index  used  and  give  essential  in- 
formation associated  with  the  index.  Quolify  index  cal- 
culated by  the  word  "computed".  Avoid  using  expressions 
of  the  form  a  ±  b  unless  the  meaning  of  b  is  explained 
fully  immediately  following  or  in  footnote. 

(31).  Systematic  error  negligible. 
Bounds  to  imprecision  (c),  (d),  (e), 
or  (f)  stated  to  no  more  than  two 
significant  figures. 

Same  as  under  (3). 

(4).  Neither  systematic  error  nor 
imprecision    negligible.    Two  nu- 
merics indicating  bounds  to  sys- 
tematic  error   and    index  of  im- 
precision respectively 

(2)  and  (3)  above  separately  stated. 

(4').    Bounds  to  systematic  error 
and  imprecision  combined,  indi- 
cating the  likely  inaccuracy  of  the 
value 

(2)  and  (3')  above  where  the  two  components  either 
have  been  previously  described,  or  explained  im- 
mediately following  (or  in  footnote). 

(5).  Quoted  from  literature 

State  reference  and  give  author's  interpretation  of  the 
uncertainty;  add  remark  if  meaning  unknown  or 
ambiguous. 

If  rj  is  not  known,  use  the  computed  standard  deviation  s  based  on  k  measurements  as  an  estimate  of  cr,  where 

sJ  =  — —     2      (x'\  -  iy)2  ■     Thenumber(k-l)isthedegreesoffreedom  associated  with  s. 
k_1  i=l 

2  For  interpretation  see  Chapter  1,  NBS  Handbook  91,  Experimental  Statistics,  by  M.  G.  Natrella,  1963. 

3  For  details  see  Chapter  5  (straight  line),  and  Chapter  6  (multivariate  and  polynomial),  NBS  Handbook  91  . 


4  For  details  see  "Notes  on  the  use  of  propagation  of  error  formulas",  by  Harry  H.  Ku,  NBS  Journal  of  Research,  Vol. 
70C,  No.  4,  October-December,  1966. 

5  See  "Realistic  Evaluation  of  the  Precision  and  Accuracy  of  Instrument  Calibration  Systems"  by  Churchill  Eisenhart, 
NBS  Journal  of  Research,  Vol.  67C,  No.  2,  April-June,  1963,  and  "Systematic  Errors  in  Physical  Constants"  by  W.  J. 
Youden,  Physics  Today  14,  1961. 
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FIGURE  1  -  SUMMARY  OF  RECOMMENDATIONS  ON 
EXPRESSIONS  OF  THE  UNCERTAINTIES  OF  FINAL  RESULTS 


SYSTEMATIC  ERROR  AND 
IMPRECISION  BOTH  NEGLIGIBLE 

(CASE  1) 

In  this  case,  the  reported  result  should 
be  given  correct  to  the  number  of  signifi- 
cant figures  consistent  with  the  accuracy 
requirements  of  the  situation,  together  with 
an  explicit  statement  of  its  accuracy  or 
correctness. 

SYSTEMATIC  ERROR  NOT 
NEGLIGIBLE,  IMPRECISION 
NEGLIGIBLE  (CASE  2) 

(a)  Qualification  of  a  reported  result 
should  be  limited  to  a  single  quasi-absolute 
type  of  statement  that  places  bounds  on  its 
inaccuracy; 

(b)  These  bounds  should  be  stated  to  no 
more  than  two  significant  figures; 

(c)  The  reported  result  itself  should  be 
given  (i.e.,  rounded)  to  the  last  place  af- 
fected by  the  stated  bounds,  unless  it  is 
desired  to  indicate  and  preserve  such  rela- 
tive accuracy  or  precision  of  a  higher  order 
that  the  result  may  possess  for  certain  par- 
ticular uses; 

(d)  Accuracy  statements  should  be  given 
in  sentence  form  in  all  cases,  except  when 
a  number  of  results  of  different  accuracies 
are  presented,  e.g.,  in  tabular  arrangement. 
If  it  is  necessary  or  desirable  to  indicate 
the  respective  accuracies  of  a  number  of 
results,  the  results  should  be  given  in  the 

form  a  ±  b  (or  a^  j?,  if  necessary)  with  an 

appropriate  explanatory  remark  (as  a  foot- 
note to  the  table,  or  incorporated  in  the 
accompanying  test)  to  the  effect  that  the 

±  b,  or  *  j? ,  signify  bounds  to  the  errors 

which  the  a's  may  be  subject. 

(e)  The  fact  that  the  imprecision  is  negli- 
gible should  be  stated  explicity. 

NEITHER  SYSTEMATIC  ERROR  NOR 
IMPRECISION  NEGLIGIBLE  (CASE  3) 

(a)  A  reported  result  should  be  qualified 
by:  (1)  a  quasi-absolute  type  of  statement 
that  places  bounds  on  its  systematic  error; 
and,  (2)  a  separate  statement  of  its  stan- 
dard error  or  of  an  upper  bound  thereto, 
whenever  a  reliable  determination  of  such 
value  or  bound  is  available  —  otherwise,  a 
computed  value  of  the  standard  error  so 
designated  should  be  given,  together  with 
a  statement  of  a  number  of  degrees  of  free- 
dom on  which  it  is  based; 


(b)  The  bounds  to  its  systematic  error 
and  the  measure  of  its  imprecision  should 
be  stated  to  no  more  than  two  significant 
figures; 

(c)  The  reported  result  itself  should  be 
stated,  at  most,  to  the  last  place  affected 
by  the  finer  of  the  two  qualifying  state- 
ments, unless  it  is  desired  to  indicate  and 
preserve  such  relative  accuracy  or  preci- 
sion of  a  higher  order  that  the  result  may 
possess  for  certain  particular  uses; 

(d)  The  qualification  of  a  reported  result, 
with  respect  to  its  imprecision  and  syste- 
matic error,  should  be  given  in  sentence 
form,  except  when  results  of  different  preci- 
sion or  with  different  bounds  to  their  sys- 
tematic errors  are  presented  in  tabular  ar- 
rangement. If  it  is  necessary  or  desirable 
to  indicate  their  respective  imprecisions  or 
bounds  to  their  respective  systematic  errors, 
such  information  may  be  given  in  a  parallel 
column  or  columns,  with  appropriate  iden- 
tification. 

SYSTEMATIC  ERROR  NEGLIGIBLE, 
IMPRECISION  NOT  NEGLIGIBLE 

(CASE  4) 

(a)  Qualification  of  a  reported  value  should 
be  limited  to  a  statement  of  its  standard 
error  or  of  an  upper  bound  thereto,  when- 
ever a  reliable  determination  of  such  value 
or  bound  is  available.  Otherwise,  a  com- 
puted value  of  the  standard  error  so  desig- 
nated should  be  given,  together  with  a  state- 
ment of  the  number  of  degrees  of  freedom 
on  which  it  is  based; 

(b)  The  standard  error,  or  upper  bound 
thereto,  should  be  stated  to  not  more  than 
two  significant  figures; 

(c)  The  reported  result  itself  should  be 
stated,  at  most,  to  the  last  place  affected 
by  the  stated  value  or  bound  to  its  impre- 
cision, unless  it  is  desired  to  indicate  and 
preserve  such  relative  precision  of  a  higher 
order  that  the  result  may  possess  for  cer- 
tain particular  uses; 

(d)  The  qualification  of  a  reported  result 
with  respect  to  its  imprecision  should  be 
given  in  sentence  form,  except  when  results 
of  different  precision  are  presented  in  tabu- 
lar arrangement  and  it  is  necessary  or  de- 
sirable to  indicate  their  respective  impre- 
cisions, in  which  event  such  information  may 
be  given  in  a  parallel  column  or  columns, 
with  appropriate  identification. 

(e)  The  fact  that  the  systematic  error  is 
negligible  should  be  stated  explicitly. 


10 


POSTSCRIPT 


Over  the  intervening  years  since  the  publication  of 
Eisenharts  and  Ku  's  articles,  it  has  become  apparent  that  a 
few  additional  comments  may  be  useful.  It  is  equally 
apparent  that  a  complete  revision  is  neither  necessary  nor 
desirable  inasmuch  as  the  major  thrust  and  content  of  the 
articles  remain  as  valid  and  as  appropriate  as  when  first 
written.  For  this  reason,  these  comments  are  made  as  a 
postscript. 


Uncertainty  Assessments  Must  Be  Complete 

The  uncertainty  of  a  reported  value  is  meant  to  be  a 
credible  estimate  of  the  likely  limits  to  its  actual  error,  i.e., 
the  magnitude  and  sign  of  its  deviation  from  the  truth.  As 
such,  uncertainty  statements  must  be  based  on  as  nearly 
complete  an  assessment  as  possible.  This  assessment 
process  must  consider  every  conceivable  source  of 
inaccuracy  in  the  result. 

A  measurement  process  generally  consists  of  a  very 
complicated  sequence  of  many  individual  unit  operations 
or  steps.  Virtually  every  step  in  this  sequence  introduces 
a  conceivable  source  of  inaccuracy  whose  magnitude 
must  be  assessed.  These  sources  include: 

•  Inherent  stochastic  variability  of  the  measurement 
process; 

•  Uncertainties  in  standards  and  calibrated  apparatus; 

•  Effects  of  environmental  factors,  such  as  variations  in 
temperature,  humidity,  atmospheric  pressure,  and  power 
supply  voltage; 

•  Time-dependent  instabilities  due  to  gradual  and  subtle 
changes  in  standards  or  apparatus; 

•  Inability  to  realize  physical  model  because  of 
instrument  limitations; 

•  Methodology  procedural  errors,  such  as  incorrect 
logic,  or  misunderstanding  what  one  is  or  should  be 
doing; 

•  Uncertainties  arising  from  interferences,  impurities, 
inhomogeneity,  inadequate  resolution,  incomplete 
discrimination,  etc.; 

•  Metrologist  errors,  such  as  misreading  of  an 
instrument; 

•  Malfunctioning  or  damaged  apparatus; 

•  Laboratory  practice  including  handling  techniques, 
cleanliness,  etc.;  and 

•  Computational  uncertainties  as  well  as  errors  in 
transcription  of  data,  and  other  calculational  or  arithme- 
tical mistakes. 

This  list  should  not  be  interpreted  as  exhaustive,  but 
rather  as  illustrative  of  the  most  common  generic  sources 
of  inaccuracy  that  may  be  present. 

The  various  sources  of  inaccuracy  are  generally 
classified  into  sources  of  imprecision  (random 
components)  and  sources  of  bias  (fixed  offsets).  To  which 
category  a  particular  source  should  be  properly  assigned 
is  often  difficult  and  troublesome.  In  part,  this  is  because 
many  experimental  procedures  or  individual  steps  in  the 
overall  measurement  process  embody  both  systematic  and 


stochastic  (random)  elements.  (For  an  alternative 
discussion  that  questions  the  need  for  a  clear  cut 
distinction  between  random  and  systematic  components 
of  uncertainty,  see  [7].)  One  practical  approach  is  to 
classify  the  sources  of  inaccuracy  according  to  how  the 
uncertainty  is  estimated.  In  this  way,  sources  of 
imprecision  are  considered  to  be  those  components  which 
can  be  and  are  estimated  by  a  statistical  analysis  of 
replicate  determinations.  For  completeness,  the  systematic 
uncertainty  components  can  be  considered  to  be  the 
residual  set  of  conceivable  sources  of  inaccuracy  that  are 
biased  and  not  subject  to  random  variability,  and  those 
that  may  be  due  to  random  causes  but  cannot  be  or  are  not 
assessed  by  statistical  methods.  The  systematic  category 
includes  sources  of  inaccuracy  other  than  biases  in  order 
to  obtain  a  complete  accounting  of  all  sources  of 
inaccuracy  in  the  measurement  process.  Hence,  it  is 
meaningful  to  report  a  random  uncertainty  contribution, 
only  if  one  has  a  computed  statistic  for  the  magnitude  of 
its  imprecision  or  random  variation.  Many  sources  of 
inaccuracy  may  exist  consisting  of  several  components 
from  both  the  random  and  systematic  categories  and  can 
be  assessed  only  after  consideration  of  the  more  funda- 
mental processes  involved.  The  uncertainty  in  the 
calibration  of  an  instrument  with  a  standard  reference 
material,  for  example,  would  have  not  only  components 
from  the  uncertainty  in  the  standard  itself,  but  also 
uncertainty  components  arising  from  the  use  of  the 
standard  in  performing  the  calibration. 

Assessment  of  Imprecision  (Random  Uncertainties) 

Although  the  treatment  and  expressions  of  reporting 
the  imprecision  of  measurement  results  were  adequately 
covered  in  the  original  article,  a  number  of  points  are  of 
sufficient  importance  to  deserve  reemphasis. 

The  only  way  to  assess  realistically  the  overall 
imprecision  is  to  make  direct — or  preferably,  when 
possible,  indirect — replicate  determinations  [1]  and 
calculate  an  appropriate  statistic  such  as  the  standard 
error  of  the  mean.  It  is  extremely  important  to  be  definite 
on  what  constitutes  a  "replicate  determination"  because 
the  extent  to  which  conditions  are  allowed  to  vary  freely 
over  successive  "repetitions"  of  the  measurement  process 
determines  the  scope  of  the  statistical  inferences  that  may 
be  drawn  from  measurements  obtained  [2,  sec.  4.1].  When 
measurements  of  a  particular  quantity  made  on  a  single 
occasion  exhibit  closer  mutual  agreement  than 
measurements  made  on  different  occasions  so  that 
differences  between  occasions  are  indicated,  the  value  of 
the  computed  standard  error  of  the  mean  of  all  the 
measurements  obtained  by  lumping  all  of  the 
measurements  together  will  underestimate  the  actual 
standard  error  of  the  mean.  A  more  realistic  value  is 
given  by  taking  the  arithmetic  means  of  the 
measurements  obtained  on  the  respective  occasions  as  the 
replicate  determinations  and  calculating  the  standard  error 
of  their  mean  in  the  usual  way  [3,  sec.  3.5). 

In  many  situations,  it  may  not  be  possible  or  feasible 
because  of  time  and  cost  constraints  to  perform  a 
sufficient  number  of  completely  independent  determina- 
tions of  the  measurement  result.  For  results  derived  from 
several  component  quantities,  the  individual  imprecision 
estimates  must  be  propagated  to  obtain  the  imprecision  of 
the  final  result.  It  must  be  emphasized,  however,  that 
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these  estimates  of  imprecision  should  not  be  based 
exclusively  on  the  information  derived  from  just  the 
present  measurements.  Presently  derived  information 
should  be  added  to  the  information  accumulated  in  the 
past  on  the  imprecision  of  the  measurement  process.  In 
this  way,  more  realistic  and  reliable  canonical  values  of 
the  imprecision  statistics  may  be  established  over  time. 
Ideally,  every  major  step  or  component  of  the 
measurement  process  should  be  independently  assessed. 
This  would  include  not  only  the  variability  inherent  in 
the  particular  measurement  of  concern,  but  also  the 
imprecision  arising  from  corrections,  calibration  factors, 
and  any  other  quantities  that  make  up  the  final  result. 

Assessment  of  Systematic  Uncertainties 

Although  a  general  guideline  for  the  approach  to  the 
assessment  of  systematic  uncertainties  can  be  formulated, 
there  are,  unfortunately,  no  rules  to  objectively  assign  a 
magnitude  to  them.  For  the  most  part,  it  is  a  subjective 
process.  Their  magnitudes  should  preferably  be  based  on 
experimental  verification,  but  may  have  to  rely  on  the 
judgment  and  experience  of  the  metrologist.  In  general, 
each  systematic  uncertainty  contribution  is  considered  as 
a  quasi-absolute  upper  bound,  overall  or  maximum  limit 
on  its  inaccuracy.  Its  magnitude  is  typically  estimated  in 
terms  of  an  interval  from  plus  to  minus  S  about  the  mean 
of  the  measurement  result.  By  what  method  then  should 
the  magnitude  of  these  maximum  limits  be  assigned?  It 
may  be  based  on  comparison  to  a  standard,  on 
experiments  designed  for  the  purpose  [4],  or  on 
verification  with  two  or  more  independent  and  reliable 
measurement  methods.  Additionally,  the  limits  may  be 
based  on  judgment,  based  on  experience,  based  on 
intuition,  or  based  on  other  measurements  and  data.  Or 
the  limits  may  include  combinations  of  some  or  all  of  the 
above  factors.  Whenever  possible,  they  should  be 
empirically  derived  or  verified.  The  reliability  of  the 
estimate  of  the  systematic  uncertainty  will  largely  depend 
on  the  resourcefulness  and  ingenuity  of  the  metrologist. 

The  Need  for  an  Overall  Uncertainty  Statement 

Without  deprecating  the  perils  of  shorthand  expres- 
sions, there  is  often  a  need  for  an  overall  uncertainty 
statement  which  combines  the  imprecision  and  systematic 
uncertainty  components.  Arguments  that  it  is  incorrect 
from  a  theoretical  point  of  view  to  combine  the 
individual  components  in  any  fashion  are  not  always 
practical.  First,  an  approach  which  retains  all  details  is 
not  amenable  for  large  compilations  of  results  from 
numerous  sources.  And  second,  this  approach  shifts  the 
burden  of  evaluating  the  uncertainties  to  users.  Many 
users  need  a  single  uncertainty  value  resulting  from  the 
combination  of  all  sources  of  inaccuracy.  These  users 
believe,  and  rightly  so,  that  this  overall  estimate  of 
inaccuracy  can  be  most  appropriately  made  by  the  person 
responsible  for  the  measurement  result.  It  must  be 
emphasized,  however,  that  there  is  no  one  clearly 
superior  appropriate  method  for  reporting  an  overall 
uncertainty,  and  that  the  choice  of  method  is  somewhat 
arbitrary.  Several  methods  are  commonly  employed  [5,6]. 

One  method  is  to  add  linearly  all  components  of  the 
systematic  uncertainty  and  linearly  add  the  total  to  the 
imprecision  estimate.  Since  the  individual  systematic  un- 
certainties (8y)  are  considered  to  be  maximum  limits,  it 


logically  should  be  added  to  an  imprecision  estimate  at  a 
similar  confidence  level.  That  is,  for  example,  the  overall 
uncertainty  u  may  be  given  by 

q 

u  =  [tv(a)]s  +    1  8, 

where  s  is  the  computed  standard  error  based  on  v 
degrees  of  freedom,  r„(a)  is  the  Student?  value 
corresponding  to  a  two-tail  significance  level  of  a =0.05, 
0.01,  or  0.001  (depending  on  the  practice  in  the 
measurement  field  concerned),  and  8;  is  the  magnitude  of 
the  estimated  systematic  uncertainty  for  each  of  the 
identified  q  systematic  uncertainty  components.  This 
approach  probably  overestimates  the  inaccuracy,  but  can 
be  considered  as  an  estimate  of  the  maximum  possible 
limits.  For  example,  if  someone  estimated  that  five  contri- 
butions of  about  equal  magnitude  made  up  the  total 
systematic  error,  that  person  would  have  to  be  very 
unlucky  if  all  five  were  plus,  or  all  five  were  minus.  Yet, 
if  there  was  one  dominant  contributor,  it  might  be  a  very 
valid  approximation. 

Two  other  approaches  have  also  been  widely  used. 
These  methods  add  in  quadrature  all  of  the  systematic 
uncertainty  components,  and  either  add  the  resulting 
quantity  linearly  to  the  standard  error  estimate, 


s  +  V  2  8? 
7=1 

or  add  it  in  quadrature  to  the  standard  error  estimate, 


Vs2  +  I  dj 
7=1 

These  are  frequently  considered  (erroneously)  to 
correspond  to  a  confidence  level  with  P=  68%. 

In  another  method,  often  termed  the  PTB  approach  [6], 
the  component  systematic  uncertainties  are  assumed  to  be 
independent  and  distributed  such  that  all  values  within 
the  estimated  limits  are  equiprobable  (rectangular  or 
uniform  distribution)  [8].  With  these  assumptions,  the 
rectangular  systematic  uncertainty  distributions  can  be 
convoluted  to  obtain  a  combined  probability  distribution 
for  which  the  variance  may  be  computed.  This  may  then 
be  combined  in  quadrature  with  that  for  the  random 
uncertainty.  In  its  simplest  form,  the  uncertainty 
components  are  combined  to  form  an  overall  uncertainty 
by 


u  =  k  W  +  (1/3)    I  8]  , 
7=1 

where  k  is  customarily  taken  as  2  or  3.  The  above  simple 
form  is  not  appropriate  when  one  of  the  component  8y's  is 
much  larger  than  the  others;  in  such  a  case  it  will  be 
more  informative  to  keep  that  component  separate  from 
the  others  and  add  it  linearly. 
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A  Concluding  Thought 

If  there  is  one  fundamental  proposition  for  the  expres- 
sion of  uncertainties,  it  is 

The  information  content  of  the  statement  of 
uncertainty  determines,  to  a  large  extent,  the  worth  of 
the  final  result. 

This  information  content  can  be  maximized  by  following 
a  few  simple  principles: 

BE  EXPLICIT 
PROVIDE  DETAILS 
DON'T  OVERSIMPLIFY 

When  an  overall  uncertainty  is  reported,  one  should 
explicitly  state  how  the  separate  components  were 
combined.  In  addition,  for  results  of  primary  importance, 
a  detailed  discussion  and  complete  specification  of  all  of 
the  separate  uncertainty  components  is  still  required.  In 
this  way,  some  users  will  benefit  from  having  the 
metrologist's  estimate  of  the  overall  uncertainty,  while 
more  sophisticated  users  will  still  have  access  to  all  of  the 
information  necessary  for  them  to  evaluate,  combine,  or 
use  the  uncertainties  as  they  see  fit. 
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